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The MAILING DATE of this communication appears on the cover sheet with the correspondence address ~ 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

• Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
eamed patent term adjustment. See 37 CFR 1.704(b). 

Status 

1)S Responsive to comnfiunication(s) filed on 15 September 2003 . 
2a)D This action is FINAL. 2b)|^ This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex pa/te Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) 1^ Claim(s) 1-35 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) n Claim(s) is/are allowed. 

6) |E Claim(s) 22-25,27-32,34 and 35 is/are rejected. 

7) S Claim(s) 26 and 33 is/are objected to. 

8) n Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) S The specification is objected to by the Examiner. 

10) 13 The drawing(s) filed on 15 September 2003 is/are: a)^ accepted or b)n objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) 0 The oath or declaration is objected to by the Examiner. Note the attached Office Action or fomi PTO-1 52. 

Priority under 35 U.S.C. § 119 

12) 0 Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)n All b)n Some * c)^ None of: 

1 .□ Certified copies of the priority documents have been received. 

2. n Certified copies of the priority documents have been received in Application No. . 

3. n Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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3) M Information Disclosure Statement(s) (PTO-1 449 or PTO/SB/08) 5) □ Notice of Informal Patent Application (PTO-1 52) 

Paper No(s)/Mail Date . 6) □ Other: 

U.S. Patent and Trademartt Office 

PTOL-326 (Rev. 7-05) Office Action Summary Part of Paper No./Mail Date 1 1 1 72005 
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DETAILED ACTION 
Preliminary Amendment 

1. Claims 1-21 are cancelled. 

2. Claims 22-35 are added 

3. The specification has been amended with a paragraph on page. 1 , line 3. 

Specification 

1 . The disclosure is objected to because of the following informalities: In the 
paragraph on pg. 1 , line 3, please add that application 09/820396 to which this 
application claims priority to is now also Patent #6654018. 

Appropriate correction is required. 

Claim Objections 

2. Claim 26 is objected to because of the following informalities: The phrase "using 
a" is repeated twice in a row. Appropriate correction is required. 

Claim Rejections - 35 USC § 103 

3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

4. Claims 22-25, 27, 29-32, and 34 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Ezzat et al. (NPL document, "Visual Speech Synthesis by Morphing 
Visemes", herein referred to as "Ezzat") in view of Jiang et al. (NPL document, "Visual 
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Speech Analysis with Application to Mandarin Speech Training", herein referred to as 
"Jiang"). 

As per claims 22, 23, and 30, Ezzat teaches the claimed "selecting" step on top 

of 1^^ column on pg. 51 and states: 

"there are many intermediate frames that lie between the chosen viseme 
images ... Consequently, we compute a series of consecutive optical 
flowvectors between each intermediate image and its successor, and 
concatenate them all into one large flow vector that defines the global 
transformation between the chosen visemes". (emphasis added) 

And states in the abstract: 

we are able to synchronize the visual speech stream with the audio speech 
stream, and hence give the impression of a photorealistic talking face, 
(emphasis added) 

Here, the visemes represent a generic facial image that can be use to describe a 
particular sound and the flowvectors which contain visual and sound features are used 
in conjunction with the visemes. 

Ezzat does not explicitly teach the claimed "obtaining" step. Jiang teaches the 

claimed "obtaining" step by stating in the abstract: 

At each frame, region of interest is identified and 
key information is extracted. The preprocessed acoustic 
and visual information are then fed into a modular TDNN 
and combined for visual speech analysis, (emphasis added) 

states on (pg. 114, 4.2 Acoustic and Visual Input Representation, 1st paragraph): 

For acoustic data representation, we have followed 
the well-established approach to apply FFT on the Hamming 
windowed speech data to get 16 Melscale Fourier coefficients as 
input to the Acoustic input Layer. For visual data representation, 
we have performed the lip-tracking and feature points extraction 
task by applying our 2D multi-state lip shape model. Then we 
use both the color profile of the feature points on external and 
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internal boundaries and position and movement of lip boundaries 
for feature extraction using principle component analysis (PCA). 
The extracted feature vectors are then fed to the Visual Input 
Layer, (emphasis added) 

Here, the Jiang teaches feature vectors (target feature vector) and teaches of 
visual data (visual features) and acoustic information (non-visual information). It would 
have been obvious to one of ordinary skill in the art at the time of invention to combine 
Ezzat with Jiang. Jiang teaches one advantage to obtaining feature vectors in order to 
help children improve their speech pronunciation (see section 5, pgs. 114-115, 1®* 
paragraph) by providing audio-visual feedback. 

As per claims 24-25, and 31-32, Ezzat teaches the claimed "selecting ... using a 

comparison of a combination of visual features and non-visual features with the target 

feature vector" by stating on pg. 47, 2"^ col, 2"^ paragraph: 

For any input text, we determine the appropriate sequence of viseme nfiorphs 
to make, as well as the rate of the transformations by utilizing the output of the 
natural language processing unit (emphasis added) 

In order to determine the appropriate sequence, the system would have to perform a 

comparison of visual and non-visual features with a given target vector in order to 

produce the output as stated. Further, this construction process of an appropriate 

sequence of viseme morphs would require selecting candidate image samples where 

these samples could be used to transition between through transformation. 

Ezzat teaches the claimed compiling by teaching of concatenation (see quote 

from top of 1^^ column on pg. 51 above). 
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As per claim 27 and 34, Ezzat teaches the claimed first database by teaching of 
recording and collecting one image per English phoneme (bottom of 1^^ column on pg. 
47 under "Corpus and Viseme Acquisition", also see figure 2). 

Ezzat teaches the claimed second and third database by teaching of "Flow 
database" (pg. 54, 2"^ column), which contain optical flow vectors which specify 
transition data between visemes (includes visual data and includes storing non-visual 
data i.e. sound transitions). 

As per claim 29, Ezzat teaches the claimed first database in figure 2, the claimed 
second database and the claimed third database on pg. 54, 2"^* column under "Flow 
database" where this database is formed to specify visual and non-visual data between 
animation transitions (frames). 

5. Claims 28 and 35 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Ezzat in view of Jiang in further of view of Brand (NPL Document, "Voice 
Puppetry", herein referred to as "Brand"). 

As per claims 28 and 35, Ezzat does not teach the claimed limitations. 

Brand teaches the claimed "selecting ... a number of candidates" and the 

claimed "Viterbi search" by stating on the bottom half of the 1®* col on pg. 25: 

The Viterbi sequence, while most likely, may only represent a small fraction of 
the total probability mass — ^there may be thousands of slightly different state 
sequences that are nearly as likely. If this were to happen in the voice puppet, 
V would be a very poor representation of the relevant information 
in the audio, and the animation quality would suffer greatly. 
... These problems are virtually banished with entropically estimated models 
because entropy minimization concentrates the probability mass on the 
optimal Viterbi sequence, (emphasis added) 
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Brand teaches the claimed concatenation cost by stating on pg. 26, very bottom 

of 1^ col and very top of 2"*^ col: 

We quantified this with a squared error measure of divergence between 
groundtruth (x) and reconstmcted (y) facial motion vectors, weighted to 
penalize motions in the wrong direction, (emphasis added) 

It would have been obvious to one of ordinary skill in the art at the time of invention to 
combine Brand with the combinable system of Ezzat with Jiang. Brand teaches the 
advantage of using an optimal Viterbi sequence with a large number of state sequences 
(candidates) to reduce the size to the most optimal ones in order to remove poor 
animation quality (1^' col on pg. 25 see quote above). 

Allowable Subject Matter 

6. Claims 26 and 33 are objected to as being dependent upon a rejected base 
claim, but would be allowable if rewritten In independent form including all of the 
limitations of the base claim and any intervening claims. 

Conclusion 

7. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. Georgiev et al. (US Patent 6285794). 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Daniel F. Hajnik whose telephone number is (571) 272- 
7642. The examiner can normally be reached on Mon-Fri (8:30A-5:00P). 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Ulka J. Chauhan can be reached on (571) 272-7782. The fax phone 
number for the organization where this application or proceeding is assigned is 571- 
273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more infonnation about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 
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